1 link tagged with all of: arxiv + lsa + sparse attention + long context + latent sparse attention
Click any tag below to further narrow down your results
Links
This arXiv paper proposes Latent Sparse Attention (LSA), which clusters keys in a learned latent space so queries score cluster hierarchies rather than every key, aiming to make attention cheap at over 1M tokens. The authors report speedups over full and original sparse attention at 10M tokens while keeping accuracy close to the baseline.
- Full attention becomes too costly above 1M tokens, and existing sparse indexers either remain quadratic or lose accuracy at that scale.
- LSA's distilled variant (D-LSA) speeds up one attention layer at 10M tokens by 22.7x in prefill and 4.6x in decode over the original sparse attention, with similar accuracy.
- The native variant (N-LSA), trained only on next-token loss, matches full attention accuracy with a 1.6x per-layer training speedup at 128k tokens.